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ABSTRACT 

This study explored the views of pre-service teachers regarding the indicators of information and 
communication technologies (ICT) at Turkish education faculties. A cross-sectional survey design was 
implemented with graduating students enrolled in Turkish education faculties. A combination of stratified 
random sampling and systematic sampling was implemented. Turkish education faculties were subdivided into 
six groups based upon the knowledge of how each institution stood relative to selected stratifying variables, and 
a sample was drawn from each group randomly. The data collection tool was administered to 2515 graduating 
students at those education faculties. It was found that participants criticized the current situation of ICT 
indicators in their institutions. Perceptions differed with regard to different departments, gender and frequency 
of ICT use for instructional purposes whereas they did not vary with regard to income, PC experience, and 
having a PC at home. Implications and suggestions for further research were provided. 
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INTRODUCTION 

Skills regarding information and communication technologies (ICTs) have gained incremental importance for 
education, employment and communication in recent years. ICTs have become significant tools to access 
information, educate individuals and conduct interactive instructional activities regardless of time and location 
(Mobbs, 2002). According to the final report of the World Summit on the Information Society (WSIS, 2003), 
which was signed by 175 countries, it was recommended that developing countries should be supported to make 
progress in access to ICTs and distance learning opportunities at a lower cost, so that all individuals pursue a 
sustainable progress to create an information society. Such comprehensive decisions stem from the research 
findings maintaining that organizations with high-level ICT integration have incremental productivity 
(Campbell, 2001). ICTs are considered to provide a more creative, innovative, entertaining and colorful 
atmosphere in comparison to face-to-face instructional endeavors. In this regard, the need to equip individuals 
with skills to use ICTs effectively and responsibly presents an enonnous challenge to educators, since they are 
supposed to provide learners with relevant, up-to-date and high-quality technology experience before learners 
emerge into the employment world (Gibson, O’Reilly, & Hughes, 2002). 

ICTs are dynamic in nature, so are the skills regarding ICT use. UNESCO (2002) lists four competencies 
regarding ICT integration in teacher training as (1) Content and Pedagogy, (2) Collaboration and Networking, 
(3) Technical Issues, and (4) Social Issues. Akbulut, Kesim and Odaba§i (2007) scrutinize the subtitles of each 
competency through the help of the Odaba§i et al. (2006) study. More specifically, Content and Pedagogy 
indicators involve (a) Teaching-Learning Methods and (b) ICT in the Curriculum; Collaboration and 
Networking indicators involve (a) Professional Development and (b) Learning Communities; Social Issues 
involve (a) Health, (b) Ethics, (c) Policies, and (d) Special Needs; and finally Technical Issues involve (a) 
Infrastructure, (b) Ease of Use, (c) Access, and (d) Technical Assistance. A detailed description of each 
competency may be found in UNESCO (2002) and sub-competencies can be examined in Akbulut et al. (2007). 
As a follow-up research, Akbulut (2009) purported to improve the measurement tool provided by Akbulut et al. 
(2007), added 34 new items to the data collection tool, and addressed the indicators of ICT integration at tertiary 
education. An exploratory factor analysis on 75 items eliminated 14 questions, extracted 11 factors explaining 64 
percent of the total variance with a very high internal consistency coefficient (a=.96). Extracted factors were 
named as E-leaming, Infrastructure, Teaching-Learning Methods, Policy, Special Education, Health, Learning 
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Communities, Ease of Use, E-interaction, Technical Assistance and Access. The current study primarily focused 
on those factors and investigated whether they vary according to several background variables. 

There have been studies focusing on different aspects of ICT integration in primary and secondary schools (El- 
Tigi, 2000; Eteokleous, 2004; Isikoglu, 2002; Kahveci, §ahin & Genp; 2011; McRae, 2001; Pompeo, 2004). 
There have also been studies regarding the use of ICTs by pre-service teachers and instructors of teacher training 
institutions (Shafiei, 2005; Toledo, 2005). However, the focus on the ICT use profiles of prospective teachers is 
satisfactory and well reported (e.g. Akptnar, 2003; Demiraslan & Usluel, 2005; $ahin, 2011) whereas 
organizational level models and analyses are slightly unconsidered except for some recent promising models 
(e.g. Akbulut, 2010; A§kar, Usluel & Mumcu, 2006; Usluel, A§kar & Ba§, 2008; Yiicel, Acun, Tarman & Mete, 
2011). In addition, comprehensive investigations were robust and informatory in K-12 settings as meticulously 
done in several recent studies (Akbaba-Altun, 2006; Aypay, 2010; Ozdemir & K 1 I 19 , 2007), but insufficient in 
higher education settings. In this regard, the current situation of educational bodies regarding ICT integration 
should be scrutinized better (ETS, 2007). 

Several studies have been conducted addressing the ICT integration process along with those addressing the 
problems confronted during integration endeavors. According to a comprehensive search we conducted recently, 
in the last decade, the number of publications on the use of computers in education was above 4600 and the 
number of those on computer based instruction was 3600, whereas relatively few studies were published on 
organizational ICT integration. To exemplify, El-Tigi (2000) administered a 60-item questionnaire along with 
open-ended questions to 142 undergraduate students. It was revealed that insufficient motivation, infrastructure, 
PC skills and time constraints prevented participants from implementing ICT tools whereas positive guidance, 
quality content, rich materials, ease of access and ease of communication facilitated ICT use. Smith and 
Robinson (2003) provided a new perspective for technology integration into curriculum, and suggested that 
collaborative cohorts might be used for successful integration, which could be evaluated within the framework of 
collaboration and networking in the current study. Pompeo (2004) described successful ICT integration 
endeavors and identified factors necessitating ICT integration. Four educational institutions with above-average 
ICT infrastructure were investigated through qualitative methods. Findings suggested that resources to sustain 
powerful infrastructure carried utmost importance for successful integration. It was also indicated that initial 
steps should be taken by the institution before ideal integration endeavors were realized. Among these steps were 
administrative policies and responsibilities, development of infrastructure, communication within the 
organization, and arrangements in the curriculum to address current needs. In addition, constant professional and 
technical developments of instructors were reported to carry importance. 

Recent studies addressing teacher perspectives on integrating ICTs into instruction (Toledo, 2005) and on 
teachers’ integration of ICTs into classroom practice (Hennessy, Ruthven, & Brindley, 2005) implied that 
teachers needed to develop new strategies for mediating ICT supported activities. After examining ICT use 
approaches found in teacher training, Jung (2005) suggested that ICTs could change the ways teachers teach. If 
such a transformation in teaching and learning endeavors was not realized, pre-service teachers might not find 
sufficient opportunities to experience ideal ICT implementations for instructional purposes. This argument was 
supported by Barton and Haydn (2006) indicating that pre-service teachers were influenced by their role models. 
More specifically, modeling of ICT by the mentor was considered vital. Similarly, Mueller, Wood, Willoughby, 
Ross and Specht (2008) indicated that positive learning experiences with ICTs had an impact on successful 
integration. In addition to positive experiences, significant variables influencing integration included teacher’s 
comfort with computers; beliefs supporting the use of computers as an instructional tool; training; motivation; 
support; and teaching efficacy. 

Poor classroom environments and lack of or limited availability of equipment to realize ICT-integrated lessons 
were reported as significant barriers to ICT integration in many studies (Akbaba-Altun, 2006; Brill & Galloway, 
2007; Clarke, 2007; Gokta§, Yildinm & Yildinm, 2008; Odaba§i, 2000; Ololube, 2006). Infrastructure is 
primarily an administrative problem. Strong infrastructure should be followed by equal access opportunities for 
all, precautions to facilitate ease of use, and employment of technical staff to assist users. Still, the habit of 
integrating ICTs into classrooms by transforming old-fashioned teaching endeavors to new technology settings, 
and ignoring the unique contributions of ICTs were reported as common problems (Knight, Knight, & Teghe, 
2006). Instructors mostly deal with their publications rather than quality instruction - an issue which is not 
supported through tangible awards in Turkey sufficiently. In this regard, rewarding quality instruction through 
ICTs might be a plausible solution for effective integration as indicated in several recent studies (Brill & 
Galloway, 2007; Del Favero & Hinson, 2007; Liu & Huang, 2005). In addition, developing communities of 
practice among the participants of the integration process can facilitate both ICT integration and professional 
development endeavors (Hodgkinson-Williams, Slay, & Sieborger, 2008). 
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Fortunately, there have been studies in Turkey addressing social issues of ICT integration including health 
(Odaba§i & Eri§ti, 2008), ethics (Akbulut, Odaba§t, & Kuzu, 2008a; Akbulut, $endag, Birinci, Kiltqer, §ahin & 
Odaba§t, 2008b; Akbulut, Uysal, Odaba^t & Kuzu, 2008c; Akbulut, $ahin & Eri§ti, 2010), special education 
(Girgin, Kiyict & Tanyeri, 2008; Odaba§t, £uhadar & Kuzu, 2008) and policy issues (Akbaba-Altun, 2006). The 
primary purpose of the current study was to investigate the current situations of Turkish education faculties with 
regard to the indicators of ICT integration through pre-service teachers’ viewpoints. In addition, pre-service 
teachers’ perceptions regarding the ICT integration level of their education faculties were also investigated with 
regard to several background variables determined according to recent studies (Akbulut et al., 2008c; Alampay, 
2006; Campbell, 2001; Hartley, 2007; Hohlfeld, Ritzhaupt, Baron & Kemker, 2008; Ilomaki & Rantanen, 2007; 
Rodriguez, 2006; Underwood & Szabo, 2003; Vekiri & Chronaki, 2008; Wainer et al., 2008) which were gender, 
department, family income, PC experience, having a PC at home, and frequency of ICT use for instructional 
purposes. 

METHODS AND PROCEDURES 
Sampling 

The study resorted to a cross-sectional survey design whose population consisted of pre-service teachers enrolled 
in Turkish education faculties. Turkish universities showed a heterogeneous distribution in terms of the degree of 
academic achievement and infrastructure. In this regard simple random sampling was considered ineffective. A 
combination of stratified random sampling and systematic sampling was applied. To reduce the possibility that 
the sample might turn out to be unrepresentative of the population, the education faculty population was first 
described and listed according to some quality criteria (e.g. number of students per instructor, number of indexed 
articles per instructor, university entrance exam ranks, etc.). Then, the list was subdivided into six parts based 
upon the knowledge of how each faculty stood relative to stratifying variables, and a sample was drawn 
randomly from each part consisting of six education faculties. 

Of 5371 last year education faculty students in these six universities, 2627 participants voluntarily participated in 
the study which constituted a response rate of 49 percent. However, 112 questionnaires (4 %) were eliminated 
since those participants filled in the questionnaire with a monotonous pattern (e.g. marking all items as 5 or 1), 
or left at least half of the items empty, or filled in their personal information form but left other items empty. 
After this elimination, the number of valid questionnaires was 2515 (47 %). Of these participants, 1595 (63.4 %) 
were females and 881 (35 %) were males whereas 39 (1.6 %) did not indicate their genders. 

Data Collection and Analysis 

A personal information form and 75 items were administered to graduating students at six education faculties. 
Items were developed through extensive literature review, focus group interviews with ICT instructors and PhD 
students, expert panels with scholars in the field and pilot implementations. Items were rated according to a five- 
point scale. Scale development processes were reported in Akbulut (2009), who eliminated 14 dysfunctional 
items through exploratory factor analysis. The final scale included 11 factors sheltering 61 items, which 
explained 64 percent of the total variance. The internal consistency coefficient of the tool was 0.96. Name of the 
factors and internal consistency coefficients were as follows: E-leaming (a=.93), Infrastructure (a=.89), 
Teaching-learning Methods (a=.86), Policy (a=.89), Special Education (a=.88), Health (a=.87), Learning 
Communities (a=.83), Ease of Use (a=.88), E-interaction (a=.85), Technical Assistance (a=.84), and Access 
(a=.78). 


Exploratory and confirmatory factor analyses on the data were reported by Akbulut (2009). Descriptive statistics 
and parametric comparisons among the levels of background variables were reported in the current study. 
Descriptive values regarding each indicator were given, and these values were compared with the neutral value 
(i.e. 3 out of 5) through one-sample t-tests. Comparisons regarding background variables with more than two 
levels were conducted through one-way between-groups ANOVAs, whereas comparisons regarding 
dichotomous variables like gender and having a PC at home were conducted through independent-samples t- 
tests. The relationship between participant scores on the current scale and PC experience was examined through 
correlation coefficients. 

RESULTS 

The overall evaluation 

First, descriptive statistics regarding each indicator were calculated. The mean of the whole scale was 2.16 (SD= 
0.57). None of the factor means was close to the medium value (i.e.3). One-sample t-tests revealed that all 
factors and the average of all questions were significantly lower than a medium value of 3 at a probability value 
below 0.001. Sorting factors from the highest through the lowest provided the following order: Learning 
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Communities (V=2.79; SD=0.8), Teaching-learning methods (A=2.78; SD=0.76), Health (A=2.75; SD=0.89), 
Access (X=2.26; SD=0.91), Infrastructure (A=2.16; SD=0.86), Technical Assistance (A=2.09; SD=0.9), E- 
leaming (A=2.02; SD=0.92), E-interaction (A=1.89; SD=0.88), Policy (A=1.78; SD=0.76), Ease of Use 
(A=1.76; SD=0.83) and Special Education (A=1.45; SD=0.68). 

Gender 

After the exclusion of 39 participants (1.6 %) who did not indicate their genders, 881 male and 1595 female 
participants were compared in terms of the average score. Independent-samples t-test revealed that average of 

males’ evaluations regarding ICT indicators (X=2.21; SD=0.6) was significantly higher than that of females 

(X=2.\3‘, SD=0.55) at a statistically significant level (p<0.001). Further comparisons for each indicator 
revealed that males’ evaluations were significantly more positive than those of females in terms of E-leaming, 
Policy, Special Education and Technical Assistance. Females’ evaluations were more positive in terms of 
Learning Communities. Other indicators did not differ between males and females. All significant differences 
were at a probability value below 0 . 01 . 

Department 

Participants’ average values on the scale with regard to their departments were examined through a one-way 
ANOVA. Descriptive statistics are provided in Table 1. 


Table 1. Descriptive statistics of each department 


Department 

Science and 
mathematics 

Social sciences 

Turkish language 

Foreign languages 

Computer sciences 

Educational 

sciences 

Fine arts 

Primary education 

Pre-school 

education 

Total 

n 

601 

290 

306 

366 

154 

92 

43 

515 

139 

2506 

X 

2,23 

2,24 

2,01 

2,09 

2,37 

2,02 

1,91 

2,10 

2,33 

2,16 

SD 

0,55 

0,64 

0,50 

0,53 

0,57 

0,47 

0,66 

0,58 

0,60 

0,57 


As shown in the table, departments of fine arts education, educational sciences and Turkish language education 
had the lowest means. To see whether the differences among departments were statistically significant, a one¬ 
way ANOVA was conducted, which revealed an F value of 12.63 with a corresponding significance below 
0.001. Since the homogeneity of variance assumption was not met, multiple comparisons were conducted 
through Tamhane’s T2. Mean differences are provided and significant differences are marked in Table 2. As 
indicated in the table, departments of computer science education and preschool education had significantly 
higher means than the departments of Turkish language education, foreign languages education, educational 
sciences, fine arts education and primary school education. Science and mathematics education had higher 
means than Turkish language education, foreign languages education, educational sciences and primary 
education. Social science education had higher means than Turkish language education, foreign languages 
education, and educational sciences. In brief, departments of Turkish language education, educational sciences 
and fine arts education had lower means which created significant differences in multiple comparisons. 


Table 2. Multiple comparisons and mean differences among departments 


Group 

Science and 
mathematics 

(A) 

Social sciences 

(B) 

Turkish 
language (C) 

Foreign 
languages (D) 

Computer 
sciences (E) 

Educational 
sciences (F) 

Fine arts (G) 

Primary 
education (H) 

Pre-school 
education (I) 

A 

-0,01 

0,23*“ 

0,14“ 

-0,14 

0,21“ 

0,32 

0,13“ 

-0,1 

B 


0,24“* 

0,15* 

-0,13 

0,23“ 

0,34 

0,14 

-0,08 

C 



-0,09 

-0,37*“ 

0,01 

0,1 

-0,09 

-0,33*“ 

D 




-0,28“* 

0,08 

0,102 

-0,18 

-0,24*** 

E 





0,36*” 

0,47*** 

0,28*” 

0,04 

F 






0,11 

-0,08 

-0,32*** 

G 







-0,19 

-0,42* 

H 








-0,23“ 
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Family Income 

Of 2515 participants, 53 (2 %) did not indicate their family income. About 45 percent of the participants earned 
400 to 800 USD per month; 25 percent earned 801 to 1200 USD; 9 percent earned 1201 to 1600 USD. Seven 
percent earned higher than 1600 dollars whereas 12 percent earned lower than 400 dollars per month. To 
compare different family income groups in terms of the average of the scale, a one-way between-groups 
ANOVA was conducted. An F value of 0.91 with a corresponding significance of 0.46 revealed that participants 
from different socioeconomic groups did not differ in terms of their responses to the scale. 

Instructional PC Use 

Participants were asked about the frequency of instructional ICT use. Means of each ICT use group for the 
current scale are provided in Table 3. 


Table 3. Descriptive statistics of each instructional ICT use group 



Everyday 

2-3 times a 

week 

Once a 

week 

1 -2 times a 

month 

1-2 times a 

semester 

Never 

Total 

N 

380 

820 

532 

449 

291 

26 

2498 

X 

2,25 

2,19 

2,15 

2,08 

2,08 

2,27 

2,16 

s 

0,61 

0,57 

0,56 

0,56 

0,52 

0,67 

0,57 


As shown in the table, those who used ICTs more often seemed to have higher means than those who used them 
less frequently. The F value of 5.42 with a corresponding probability value below .001 indicated that 
instructional ICT use had an effect on averages. Multiple comparisons through Tamhane’s T2 are provided in 
Table 4. 


Table 4. Multiple comparisons and mean differences among levels of instructional ICT use 


Group 

Everyday 

(A) 

2-3 times a 
week (B) 

Once a 
week (C) 

1 -2 times a 
month (D) 

1 -2 times a 

semester 

(E) 

Never (F) 

A 


0,06 

0,1 

0,16*’* 

0,17*** 

-0,02 

B 



0,04 

0,11* 

0,11* 

-0,08 

C 




0,06 

0,07 

-0,12 

D 





0,01 

-0,18 

E 






-0,19 

F 








As shown in the table, the averages of those who used ICTs everyday or 2-3 times a week were significantly 
higher than those who used it 1-2 times a month or 1-2 times a semester. That is, those who used ICTs for 
instructional purposes had more positive opinions than those who used them less. Interestingly, those who never 
used ICTs for instructional purposes showed a different pattern than other ICT use groups and had the highest 
mean. 

Computer Experience 

Participants’ PC experience ranged from 1 through 15 years. The average PC experience was 5.66 years with a 
standard deviation of 2.3. Neither parametric, nor non-parametric correlations between the scale average and the 
PC experience were significant (p > 0.194). In addition to PC experience, 1557 participants (63 %) who had a 
PC at home / dormitory and 929 participants (37 %) who did not have a PC were compared through an 
independent-sample t-test, which revealed that the averages were almost equal (p = 0.993). 

CONCLUSION AND DISCUSSION 

Findings of the current study showed similarities with some previous studies. For instance, the perceived 
negative picture regarding the ICT indicators supported the argument of Giilbahar (2008). That is, teacher 
training programs did not facilitate the effective integration and use of ICTs for instructional purposes 
sufficiently. ICT integration in instructional activities was considered ineffective (Demirarslan & Usluel, 2005), 
methods to help students to make use of technology were not followed (McRae, 2001), meaningful ICT 
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experiences were considered limited (Pontier, 2005), instructors made limited use of ICTs for instructional 
purposes (Selwyn, 2007), a discrepancy between universal principles and actual classroom implementations 
were observed (Tondeur, Braak & Valcke, 2007), and programs and endeavors followed were found insufficient 
(Ajwa, 2007). Findings regarding weak infrastructure supported several previous studies as well (Akbaba-Altun, 
2006; Clarke, 2007; El-Tigi, 2000; Gokta§ et al, 2008; Giilbahar, 2008; Odaba§t, 2000; Ololube, 2006). Findings 
regarding technical assistance supported the argument of Tallent-Runnels et al. (2006) indicating that most 
universities did not have facilities to provide students and instructors with effective technical assistance. 

Perceived problems related to Teaching-Learning Methods and Technical Assistance were parallel with the 
findings of Gokta§ et al. (2008), which found similar results in primary and secondary education. As pre-service 
teachers did not experience appropriate ICT use for instructional purposes, they should not be expected to 
implement ICTs in their own classrooms (Barton & Flaydn, 2006). Negative opinions regarding Policy indicated 
that administrators were unsuccessful in implementing constructive programs and policies to improve student 
attitudes (Gay et al., 2006). The negative picture observed in terms of Special Education supported the 
arguments of Edybum (2000) and Morrison (2007), that is, current ICT implementations were far behind what 
was possible to do with them. In addition, even though creating learning communities within the society was 
regarded as an indicator of successful ICT integration (Hodgkinson et al., 2008); the average was quite low in 
Turkish education faculties. 

Differences in terms of department and gender were supported by studies addressing digital divide (Alampay, 
2006; Campbell, 2001). That is, different field experiences in departments and different life experiences 
regarding gender led to different opinions in terms of ICT integration. It was interesting that there was not any 
difference among participants with different family incomes. This finding somewhat conflicts with the 
arguments of Alampay (2006) and Hohlfeld et al. (2008) indicating an influence of socio-economic status on 
ICT related endeavors. 

The fact that men had more positive opinions regarding technical issues was expected (Tanyeri, 2008; Vekiri & 
Chronaki, 2008). However, their positive opinions with regard to E-leaming, Policy and Special Education 
should be further investigated. In addition, females found indicators of Learning Community more effective than 
males, which should be further examined. Finally, the fact that males and females did not differ in terms of 
Health indicators refuted a recent study (Odaba§i & Eri§ti, 2008). 

Abovementioned differences in terms of gender and departments can be explained through gender socialization 
and occupation socialization theories. These theories were tested in the Mason and Mudrack (1996) study and 
supported in the Akbulut et al. (2008c) study. Gender socialization theory implies that women are more likely 
than men to be socialized to obey rules (Ward & Beck, 1990). On the other hand, occupational socialization 
theory implies that individuals are similar in outlook regardless of their genders (Adam, 2000). Women’s 
positive opinions regarding Learning Communities and men’s positive opinions regarding E-leaming, Policy and 
Special Education might be explained through gender socialization theory. Differences in terms of departments, 
on the other hand, might be explained through the occupation socialization theory. 

Similar to the Czemiewicz and Brown (2005) study, ICT use levels and frequencies did not have an effect on 
ICT indicators. On the other hand, the frequency of ICT use for instructional purposes had an effect on averages. 
That is, rather than the quantity of everyday PC experience, the quantity of instructional PC experience had an 
effect on perceptions regarding ICT indicators. That is, the type of experience was quite important (Dutt-Doner, 
Allen, & Corcoran, 2006). 

The negative picture in terms of pedagogy and collaboration related indicators might stem from insufficient 
professional development (Odaba§i, 2003, 2005). This negative image might be eliminated through a reward 
mechanism focusing on quality instruction. In addition, continuous professional development activities 
addressing instructional ICT use can be helpful to improve with regard to Teaching-Learning Methods, Learning 
Communities, E-leaming and E-interaction. Providing pre-service teachers with meaningful and instructional 
ICT use experiences carries importance since they cannot be expected to implement what they did not 
experience. In addition, rather than the quantity of PC experience, type of ICT experience matters. Finally, 
administrative precautions can eliminate the negative findings observed in technical and social issues. 

Further research can administer similar data collection tools in different samples, investigate covariance errors 
among given ICT indicators in different contexts, and develop structural equation models to understand 
interrelationships among indicators. Such an approach can help scholars to determine priorities for action. 
Interesting findings can be found through administering similar data collection tools across more universities and 
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investigating the relationship between ICT integration levels and university ranks by academic performance. For 
instance, a recent university ranking has been announced online by URAP Research Laboratory at 
http://www.urapcenter.org, whose findings might be used to explore the predictive power of academic ranking 
on ICT integration or vice versa. 

The data collection tool used in the current study was quite comprehensive but somewhat insufficient since each 
indicator should be investigated with more scrutiny. In this regard, qualitative endeavors addressing the reasons 
of current findings can be helpful. In addition, pre-service teachers’ opinions should be triangulated with the 
opinions of instructors, administrators and other shareholders of the process. Similar studies might be replicated 
with pre-service and in-service teachers to see the differences and similarities as well. Finally, an identical 
replication of the current study might be helpful, since several universities have been founded, quotas of existing 
departments have been increased, expelled students have been re-invited to universities, and the university 
entrance exam has been made easier to enroll more undergraduate students. Such changes might have 
transformed the nature of the research population, which necessitates replication even within the same sample. 
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